instructional practices require field study and attention to myriad contextual
factors.  Major longitudinal studies such as High School and Beyond can go into
more detailed issues than the Common Core, but here too there are limits. For
example, such a study might be able to obtain a simple characterization of an
instructional procedure such as "used team teaching", but would be unable to
obtain detail on what division of labor, the characteristics and qualifications
of each team member, etc.

If this position is accepted, then it would be appropriate to view many of the
suggestions as ideas for a research, development, or dissemination agenda
appropriate to the National Institute of Education (or its successor units) as
well as to the research community generally.  I think that NIE should be brought
into the redesign process and explore how it might pick up on some of these
ideas, either by itself or in cooperation with NCES.

I am also concerned with the danger implicit in many of the suggestions Cor what
are program evaluation functions.  Many writers believe that a national data
system should be able to tell us "what works".  This seems like a natural
expectation, but one which I believe is doomed to frustration.  In the social
and behavioral sciences, the problems of multiple causation are so complex that
our ability to sort out the causes of particular results is very limited.  Such
knowledge as we can obtain must usually come from detailed field studies, and
even there it tends to be time and context specific.

We would do better to confine our attention to establishing good measures,
together with absolute and comparative standards, of "how we're doing", and
leave program evaluation to special studies.  I believe we can establish such
measures and standards for all kinds of variables (i.e. input, process, and
outcome varables).  Indicators of effective schools or effective teaching are
examples of measures of the quality of schooling and the quality of teaching,
respectively.  They are based on research about what factors are important in
improving student outcomes, but the data system can be used to validate the
research in only the grossest manner.  We may wish to determine whether reform A
is working in those states that have adopted tt.  We can assemble the outcome
information for those states, but even given dozens of variables and controls,
will we be able to trace the effects of the reform to the outcomes, ruling out
alternative explanations.  It seems doubtful.

Problems of Utilization.  Little attention was given to the problems of data
utilization.  There ware some suggestions made concerning the need to provide
documentation on definitions, how the data were collected, etc. and to prepare
analyses and interpretations that might reduce the number of misinterpretations
of findings.  But relative to use by policy makers, there seems to be an
underlying assumption that the date will be used for rational problem-solving in
an instrumental way.  Yet there is a considerable body of research on utlization
of knowledge which shows that that is not usually the way it happens (Charles E.
Lindblom and David K, Cohen, Usable Knowledge: Social Science and Social
Problem Solving. Yale University Press, New Haven, 1979). More often, data
serve an "enlightenment function".  Weiss sums up the argument in this way:

"Research does not solve problems; it provides evidence
that can be used by men and women of judgment in their

118 for
